Part 1: {riskmetric} & {riskassessment} - a mini series for end-to-end R package validation
Eric Milliman
Disclaimer
Any opinions expressed in this presentation and on the following slides are solely those of the presenter and do not necessarily reflect those sponsoring the work
Group of ~50 companies (mostly pharma & biotech)
Mission: Enable the use of R by the Bio-Pharmaceutical industry in a regulatory setting, where the output may be used in submissions to regulatory agencies.
Two tools: what do they do
is a framework to quantify an R package’s “risk” by assessing several meaningful metrics designed to evaluate package development best practices, code documentation, community engagement, and development sustainability.
is a full-fledged R package containing a shiny front-end that augments the utility of {riskmetric}. The application’s goal is to provide a central hub for an organization to review and assess the risk of R packages, providing handy tools and guide rails along the way.
A primer on risk
Risk is a combination of quality and intended use and culture
Biogen has 3 levels of risk: death, rev/temp harm or no harm to patients
Quality is one way to mitigate risk
High quality software can still be high risk
Criteria to quantify risk
Sometimes “quality” is measurable! Software dev best practices dictate an R-package should have:
e.g. Number or Errors/Warnings/Notes from R CMD check
The internals of riskmetric
pkg_ref (class) and pkg_ref_cache (func)
Collects metadata from different sources
Stores raw metadata
Lazy evaluation of metadata
pkg_assess (class) and assess\_\* (function)
Tabular summary of metadata
e.g. Number or Errors/Warnings/Notes from R CMD check
pkg_metric (class) and metric_score (func)
Per assessment score
[0,1] bound
Multiple metrics per assessment possible
The internals of riskmetric
pkg_ref (class) and pkg_ref_cache (func)
Collects metadata from different sources
Stores raw metadata
Lazy evaluation of metadata
pkg_assess (class) and assess\_\* (function)
Tabular summary of metadata
e.g. Number or Errors/Warnings/Notes from R CMD check
pkg_metric (class) and metric_score (func)
Per assessment score
[0,1] bound
Multiple metrics per assessment possible
pkg_score (class/func)
Summary of metric scores
[0,1] bound
Can customize weights of custom metrics
Things to consider when using {riskmetric}
Package source
Not all metrics are available for all types of package sources
`pkg_ref()` can generate a list of mixed source's
Things to consider when using {riskmetric}
Package source
Not all metrics are available for all types of package sources
`pkg_ref()` can generate a list of mixed source's
Missing information
Metric only implemented for pkg_ref’s of specific source type
e.g. code coverage only available for pkg_ref.pkg_source
Parsing error – generally handled by {riskmetric}
Metric not expected because metadata missing
e.g. package does not have bug_reporting_url. Would not expect to have compute a bug closures over the last 30 days.
Things to consider when using {riskmetric}
Package source
Not all metrics are available for all types of package sources
`pkg_ref()` can generate a list of mixed source's
Missing information
Metric only implemented for pkg_ref’s of specific source type
e.g. code coverage only available for pkg_ref.pkg_source
Parsing error – generally handled by {riskmetric}
Metric not expected because metadata missing
e.g. package does not have bug_reporting_url. Would not expect to have compute a bug closures over the last 30 days.
Metric weights
Weights are applied to all pkg_metric regardless of source type.
# A tibble: 5 × 5
pkg_ref version remote_checks r_cmd_check size_codebase
<lst_f_p_> <chr> <pkg_scor> <pkg_scor> <pkg_scor>
1 accrual<source> 1.3 NA NA 0.25510204
2 dplyr<install> 1.1.2 NA NA NA
3 tools<install> 4.2.0 NA NA 0.01213789
4 arules<cran_remote> 1.7-6 1 NA NA
5 limma<bioc_remote> 3.56.2 NA NA NA
Current Roadmap
Increase ease of use
Convenient wrapper functions
helpful messaging
Cleaner reporting/output
Current Roadmap
Increase ease of use
Convenient wrapper functions
helpful messaging
Cleaner reporting/output
Completeness
# A tibble: 18 × 8
generic default pkg_ref pkg_remote pkg_install pkg_source pkg_bioc_remote
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 assess_new… NA 1 1 NA NA NA
2 assess_has… NA 1 NA NA NA NA
3 assess_siz… 1 NA NA 1 1 NA
4 assess_has… 1 NA NA NA NA NA
5 assess_exp… NA NA 1 1 1 NA
6 assess_rev… 1 NA NA NA NA NA
7 assess_dow… NA 1 NA NA NA NA
8 assess_dep… 1 NA NA 1 1 1
9 assess_r_c… 1 NA NA NA 1 1
10 assess_rem… 1 NA NA NA NA 1
11 assess_exp… 1 NA NA 1 1 NA
12 assess_has… NA 1 NA NA NA NA
13 assess_cov… 1 NA NA NA 1 NA
14 assess_las… 1 NA NA NA NA NA
15 assess_lic… 1 NA NA NA NA NA
16 assess_has… 1 NA NA NA NA NA
17 assess_has… 1 NA NA NA NA NA
18 assess_has… 1 NA NA NA NA NA
# ℹ 1 more variable: pkg_cran_remote <dbl>
Current Roadmap
Increase ease of use
Convenient wrapper functions
helpful messaging
Cleaner reporting/output
Completeness
Consistency in source -> assessment -> metric
Chain sources to increase metric coverage for analysis
Current Roadmap
Increase ease of use
Convenient wrapper functions
helpful messaging
Cleaner reporting/output
Completeness
Consistency in source -> assessment -> metric
Chain sources to increase metric coverage for analysis
3rd party metrics
Assessments/Metrics that are executed only if package dependencies are installed
oysteR, srr, autotest, pkgnet, valtools
ad hoc assessments and/or metrics
Current Roadmap
Increase ease of use
Convenient wrapper functions
helpful messaging
Cleaner reporting/output
Completeness
Consistency in source -> assessment -> metric
Chain sources to increase metric coverage for analysis
3rd party metrics
Assessments/Metrics that are executed only if package dependencies are installed
oysteR, srr, autotest, pkgnet, valtools
ad hoc assessments and/or metrics
Cohorts – collections of packages
A set of packages (e.g. tidyverse)
An environment (base, priority, plus packages required by the business)
